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Abstract 

An example of employing the Delphi technique for curriculum improvement at 
a Canadian university is described. This technique was used to identify essential 
course components to be included in a revised Office Administration program. 
In addition to providing the necessary information for updating and improving 
the curriculum, the Delphi technique allowed the department to receive infor¬ 
mation from and educate guidance counsellors and future employers with 
regard to the proposed changes in the program. The Delphi technique made it 
possible to obtain opinions from five diverse groups at relatively low cost and 
effort. The anonymity that the Delphi method provides helped to ensure that 
responses reflected accurately the opinions of the respondents. A number of 
areas of study and special competencies and skills were identified as being 
essential to a program designed to meet the needs of the automated workplace. 
The study determined that the program should encompass a much larger compo¬ 
nent in office technology than existed previously and that the focus of the pro¬ 
gram should be on information management rather than office administration. 
Along with the methodology, the advantages and disadvantages of using the 
Delphi technique for improving the quality of the curriculum are discussed. 
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Resume 

L’article ddcrit un exemple de l’emploi de la technique Delphi dans une 
university canadienne en vue d’amdliorer un programme d’dtudes. L’emploi de 
cette mdthode a permis d’identifier les composants essentiels des cours & inclure 
dans un programme d’administration de bureau rdnovd. En plus de foumir les 
donndes ndcessaires h la mise & jour et & l’amdlioration du programme, la 
technique Delphi a rendu possible un echange d’information permettant 
notamment au departement d’dduquer les conseillers d’orientation et les futurs 
employeurs quant aux modifications proposdes pour le programme. La 
technique Delphi a permis de consulter cinq groupes diffdrents pour un cout et 
un effort relativement moderns. L’anonymity garantie par les procedures de la 
mdthode a permis d’assurer que les reponses refletaient fideiement les opinions 
des rdpondants. Plusieurs domaines d’dtudes et plusieurs competences et savoir- 
faire specifiques ont ete identifies comme essentiels it un programme visant a 
repondre aux besoin d’un lieu de travail automatise. L’etude a determine que le 
programme devrait comprendre un composant en etude de bureautique 
beaucoup plus etendu que celui qui existait auparavant et qu’il devrait etre 
davantage centre sur la gestion de l’information que sur l’administration de 
bureau. L’article presente la methodologie et discute les avantages et 
ddsavantages de la technique Delphi pour la revision des programmes. 


With the recent emphasis on quality of education (Conrad & Blackburn, 1985; 
Smith, 1991; Bogue & Saunders, 1992), it is important to identify useful meth¬ 
ods for improving the curriculum. A procedure which has been used successful¬ 
ly for this purpose in the United States is the Delphi technique (Uhl, 1990). This 
method has been used in Canada as a forecasting tool and as a method of 
obtaining convergence of opinion (Nadeau et al., 1992), but rarely has it been 
reported in publications as being used for curriculum improvement. 

An objective of the Delphi technique is to obtain convergence of opinion 
without bringing individuals together in face-to-face meetings. This objective is 
usually achieved by having the participants complete a series of questionnaires 
interspersed with controlled opinion feedback. This mode of controlled interac¬ 
tion among the respondents not only leads to savings in time and money, but 
also permits independent thought among participants and assists them in the 
gradual formation of a considered opinion. It has the added advantage of ensur¬ 
ing participants’ anonymity. In contrast, direct confrontation, as experienced in 
faculty or committee meetings, often results in hasty offering of preconceived 
notions, the inclination to close one’s mind to novel ideas, a tendency to defend 
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previously taken stands, and a tendency to be influenced by the persuasively 
stated opinions of others. 

The general procedure for the Delphi technique is as follows: 

1. The participants are asked to list their opinions on a specific 
topic such as curriculum revision or planning priorities. 

2. The participants are then asked to evaluate the total resulting list 
by a criterion such as importance, chance of success, and so on. 

3. The participants receive the list and a summary of responses to 
the items. If the participants are in the minority, they are asked 
to revise their opinions or indicate their reasons for remaining in 
the minority. 

4. The participants again receive the list, an updated summary, 
minority opinions, and another chance to revise their opinions. 

The use of the Delphi technique may be warranted if any or all of the fol¬ 
lowing conditions exist: (1) the resolution of a problem can be facilitated by the 
collective judgments of one or more groups; (2) those groups providing judg¬ 
ments are unlikely to communicate adequately without an intervening process; 
(3) the solution is more likely to be accepted if more people are involved in its 
development than would be possible in a face-to-face meeting; (4) frequent 
group meetings are impractical because of time, distance, and so forth; and (5) 
one or more groups of participants are more dominant than another (Uhl, 1983). 

The following describes the employment of the Delphi technique to 
improve the curriculum of an office administration program in a relatively small 
university (approximate FTE of 4000) in Nova Scotia. 

Prior to this Delphi study, the faculty of the Department of Office 
Administration had been meeting to develop a revised program which it hoped 
would better serve the needs of the marketplace. As it developed a proposed 
new program, the faculty thought it would be beneficial to obtain from different 
groups reactions to the proposed changes and suggestions for additional 
changes. Several groups from which reactions would be helpful were identified. 
These included the department’s advisory board, graduates of the department 
since 1980, students presently enroled in the department, a selected group of 
Atlantic Canadian businesses, and high school guidance counsellors in the 
province. These five groups were selected either because of their unique posi¬ 
tions as stakeholders in the proposed curriculum revisions or because they were 
considered to be “experts” who would very likely contribute useful and timely 
information. The fact that all of these groups were considered to have a valuable 
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contribution to make even though they were, to some extent, widely divergent, 
necessitated the use of a method which might provide adequate communication 
among the groups in the form of feedback of opinions of other respondents. 
Meetings either within or between these groups were not practical because of 
time, geographic and fiscal constraints. Furthermore, had it been feasible to 
bring these groups together, it is likely that certain groups would have tended to 
dominate, as a result of their greater experience, maturity, and positions in the 
business community. For these reasons, therefore, the Delphi technique was 
selected as the best method for obtaining information, gaining the greatest 
degree of participation and achieving some convergence of opinion among 
these different groups. 


Method 


Subgroups 

The five subgroups included all 12 members of the department’s advisory 
board, 60 representatives from Atlantic Canadian businesses, all 220 Office 
Administration Department degree graduates from 1980 to the present. Nova 
Scotia high school guidance counsellors, and students presently enrolled in the 
department’s programs. The Eastern Canadian businesses were chosen on the 
basis of size as well as prominence in their communities and their industries. 
The 54 guidance counsellors were from the largest high schools in the province. 
Returning students in good academic standing who were enrolled in the pro¬ 
gram above first year level were selected. 

Questionnaire Development 

In developing the first questionnaire, material was first reviewed from uni¬ 
versities which had been involved in Office Administration curriculum revi¬ 
sions. With this information as a background, the first-round questionnaire was 
developed in consultation with the department. This questionnaire was pretested 
on a graduate class in Education composed of guidance counsellors, and on 
some Office Administration faculty and three members of the business commu¬ 
nity. The questionnaires for rounds two and three were developed on the basis 
of the responses received to this first questionnaire. The second and third ques¬ 
tionnaires reflected the responses of the previous rounds. Consistent with the 
Delphi procedure, these questionnaires were designed to give the respondents as 
much feedback as possible on the responses of the group to each question in the 
preceding questionnaire. Table 1 shows examples of types of questions in the 
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three rounds of questionnaires. The questions from rounds two and three show 
how feedback on responses of other participants was given to respondents to 
assist them in their decision-making. The Round One questionnaire contained 
more open-ended questions, as opinions were sought concerning components 
that should be included in future rounds. The subsequent rounds then contained 
more questions using Likert-type scales. 

Procedure 

Three rounds of questionnaires with covering letters (available from the senior 
author) were sent over a period of eight months. Before the first mailing, letters 
were sent to the superintendents of every school district involved, asking if any 
objected to a polling of the opinions of guidance counsellors in their districts. 
Some superintendents responded with written permission. None refused permis¬ 
sion. After the first round, the second round of questionnaires was sent to busi¬ 
ness and student subgroups who had responded to the first round. The guidance 
counsellors had a very low rate of response in the initial round (17%) and 
expressed concern with regard to their inadequate knowledge of the program 
and its graduates. Therefore this whole group was sent a revised cover letter to 
explain their role in the study, along with the second questionnaire. In addition, 
the second round was sent to all graduates who had responded to the first round, 
as well as to an additional 15 selected from the list of those who had been sent 
the first questionnaire and had not replied. (Before selecting these 15, names of 
all those graduates whose questionnaires had been returned as undeliverable 
were eliminated from the original list.) These additional graduates were added 
to the second round because of the relatively low response (32%) of graduates 
to the first round. The remaining three groups were considered to have an 
acceptable rate of return after the first round. It was recognized at the outset that 
the graduate population is a mobile one and that many of the addresses on 
record at the University would be out of date. Although every effort was made 
to acquire the most recent addresses, a large group of graduates was sent the 
first round in order to compensate for an anticipated low rate of return. The 
Advisory Board was not asked to complete the second questionnaire as they 
were involved in discussions with the faculty at the time and were providing 
feedback and recommendations. This group, therefore, received only the first 
and third rounds. 

The third-round questionnaires were sent to the same people to whom the 
second round had been sent, with the exception of those whose questionnaires 
had been undeliverable during the second round and the addition of the 
Advisory Board. 
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Table 1 

Sample Questions from Three Rounds 


Round Question 

One 2.(a) In order to develop courses that will prepare the type of grad¬ 

uate competent in the areas identified by you in Question One, 
please rate the following competencies to indicate, by circling the 
appropriate number, the degree of competence which you consider 
necessary for students in this program. 

Very competent Fairly competent Not Necessary 

Keyboarding 1 2 3 

Training skills 1 2 3 

Records management 1 2 3 

2.(b) Please add competencies that you think should have been 
included in the preceding list as well as any comments you may 
have. 


Two 


2. In order to develop courses that will prepare the type of gradu¬ 
ate competent in the areas identified by the Department and by 
you in Round One, please rank the following competencies. Please 
indicate, by circling the appropriate number, the importance of 
each competency to students in this program. Underlined numbers 
indicate the majority response from Round One. If you agree with 
the previous majority response, circle this number. If you disagree, 
please circle your choice and, if you wish, give reasons for doing 
so. Note that new items have no underlined number as there was 
no previous response. For each item, please select very important 


(1), fairly important (2), 

or not important (3) 


Keyboarding 

I 

2 

3 

Reasons for choice 
if outside maioritv. 

Training skills 

I 

2 

3 


Records management 

1 

2 

3 


Work flow analysis 

1 

2 

3 
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Table 1 (cont’d) 

Three In order to develop courses that will prepare the type of graduate 
competent in the areas identified (both by the Department and by 
you) in Question One, please rate the following competencies. 
Please indicate, by circling the appropriate number, the impor¬ 
tance of each competency to students in this program. Underlined 
numbers indicate the majority response for items in Round Two 
for which a consensus was not reached. If you agree with the pre¬ 
vious majority response, circle this number. If you disagree, please 
circle your choice. Items for which a consensus was reached are 
indicated in the appropriate columns. The right-hand column sum¬ 
marizes reasons given by some respondents for not choosing the 
majority (underlined) selection. Only reasons cited by at least two 
people are included. 


Keyhoarding 

Training skills 

Very 

Important 

Consensus 

Consensus 

Fairly 

Important 

Not 

Important 

Comments 

Records management 

1 

2 

3 

This is 
a major 
component of 
managing 
information. 

Poor 

management 
of records 
results in loss 
of time and 

Work flow analysis 

I 

2 

3 

money 


In the analysis of the data, frequencies were calculated for each element in 
all three rounds of questionnaires in order to give feedback percentages to 
respondents during the rounds and to determine if and when consensus was 
achieved. Consensus was defined as 85% agreement. This was done for the 
whole group in each round as well as for all subgroups in each round. 

In addition, the 61 respondents who had answered all three questionnaires 
were tracked, and frequencies and means and standard deviations were calculat¬ 
ed for this group. This allowed a comparison of the responses of this group with 
responses of those who completed at least one round. The comparison provided 
reassurance that congruence had not occurred as a result of those in disagree¬ 
ment with the majority not completing later rounds. 
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Results 

Table 2 shows the response rates for each subgroup in the three rounds. From a 
total of 386 questionnaires sent in the first round there were 151 returned 
(40%). Second-round questionnaires were sent to a total of 192 individuals and 
102 (53%) completed them. As previously explained, the second round did not 
include the Advisory Board. Third-round questionnaires were sent to 194 indi¬ 
viduals and 97 (50%) were completed. The larger number for round three, 
despite some attrition, is due to the inclusion of the 12 Advisory Board mem¬ 
bers who were not sent questionnaires in round two. During the three rounds, 
there was some attrition in the Business sub-group due, in part, to the very 
unstable economy during this period. This economic instability, which led to 
personnel transfers and loss of jobs, may have also contributed to attrition in the 
graduate sub-group. 

There were three main areas investigated in the three rounds of questions: 
(1) the general areas of expertise that the respondents perceive as essential to 
graduates of this program, (2) the key elements (competencies) which should be 
included in the program, and (3) the types of occupations that these graduates 
would be expected to fill. In addition, respondents were asked their opinions 
concerning the efficacy of including co-op work terms in the program and for 
input on how they thought the focus of the program should change. 

Areas of Expertise 

Consensus was reached concerning the following areas: the introduction of new 
technology; linkage of existing technology and computer systems in the work¬ 
place; managing the impact of technology on the type of work done and the way 
in which that work is done in a business environment; managing information in 
the business environment; and management of the impact of technology on peo¬ 
ple and its effect on their satisfaction, motivation, and performance. 

Competencies 

Throughout the three rounds of questionnaires, respondents were asked their 
opinions concerning the degree of competence needed by graduates in various 
specific skills and knowledge areas. The original list of 23 competencies in 
Round One was expanded to 34 in Round Two as a result of suggestions from 
respondents to the first questionnaire. Table 3 shows frequencies for responses 
concerning competencies. 

Since the response rate was not always consistent (for example, some peo¬ 
ple responded to Rounds One and Three, but not to Round Two), Table 4 shows 



Table 2 

Response Rates 

Subgroups 

Sent 

Round 1 

Returned 

Rate 

Sent 

Round 2 

Returned 

Rate 

Sent 

Round 3 

Returned 

Rate 

Advisory Board 

12 

9 

75% 

N/A 

N/A 

N/A 

12 

7 

58% 

Business 

60 

27 

45W 

25 

22 

85% 

24 

17 

~7T% 

Graduates 

220 

72 

33% 

80 

38 

48% 

75 

34 

45% 

Guidance Counsellors 

54 

9 

17% 

54 

19 

35% 

52 

15 

29% 

Students 

40 

34 

85% 

33 

23 

70% 

32 

24 

75% 

TOTAL 

386 

151 

40% 

192 

102 

53% 

195 

97 

50% 

Note: Advisory Board did not participate in Round Two. 
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Table 3 



Competencies 

R1 (N=151) 


R2 (N=102) 


R3 (N=97) 



VI 

FI 

N1 

VI 

FI 

N1 

VI 

FI 

NI 

Keyboarding 

72.8 

25.8 

1.3 

87.3 

10.8 

2.0 

C 

_ 

_ 

Proofreading 

80.8 

15.2 

4.0 

92.2 

4.9 

2.9 

C 

— 

— 

Written Communication Skills 

92.1 

7.9 

0.0 

98.0 

2.0 

0.0 

C 

— 

— 

Oral Communication 

89.4 

9.9 

0.7 

99.0 

1.0 

0.0 

C 

— 

— 

Records Management 

44.6 

53.4 

2.0 

42.6 

55.4 

2.0 

51.5 

48.5 

0.0 

Word Processing 

83.3 

16.0 

0.7 

98.0 

2.0 

0.0 

C 

— 

— 

Spreadsheets 

49.7 

47.0 

3.3 

88.1 

11.9 

0.0 

C 

— 

— 

Databases 

53.0 

45.0 

2.0 

95.0 

5.0 

0.0 

c 

— 

— 

Graphics 

28.5 

57.0 

14.6 

22.8 

75.2 

2.0 

25.8 

73.2 

1.0 

Accounting Software 

33.1 

58.3 

8.6 

27.7 

70.3 

2.0 

25.8 

70.1 

4.1 

Desktop Publishing 

29.1 

59.6 

11.3 

24.8 

71.3 

4.0 

26.8 

69.1 

4.1 

Decision Support Software 

* 

* 

* 

32.7 

57.1 

10.2 

14.4 

81.4 

4.1 

CAD/Engineering Software 

* 

* 

* 

20.2 

48.5 

31.3 

4.1 

62.9 

33.0 

Business Statistics 

17.2 

68.2 

14.6 

12.7 

81.4 

5.9 

15.5 

74.2 

10.3 

Research Methods 

23.2 

60.9 

15.9 

11.8 

79.4 

8.8 

18.6 

70.1 

11.3 

Telecommunications 

54.3 

42.4 

3.3 

88.2 

11.8 

0.0 

C 

— 

— 

Office Environment Planning/Design 

36.4 

53.6 

9.9 

25.5 

74.5 

0.0 

23.7 

70.1 

6.2 

Systems Analysis 

40.9 

51.0 

8.1 

18.8 

72.3 

8.9 

21.6 

74.2 

4.1 

Equipment Selection 

47.0 

41.1 

11.9 

34.7 

59.4 

5.9 

45.8 

49.0 

5.2 

Software Evaluation 

50.3 

40.4 

9.3 

84.2 

9.9 

5.9 

C 

— 

— 
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Table 3 (cont'd) 


Competencies 

R1 (N=151) 


R2(N=102) 


R3 (N=97) 


VI 

FI 

NI 

VI 

FI 

NI 

VI 

FI 

NI 

Training Skills 

58.7 

36.7 

4.7 

91.0 

8.0 

1.0 

C 



Computer Programming 

24.5 

43.7 

31.8 

14.9 

65.3 

19.8 

11.3 

63.9 

24.7 

Business Ethics 

65.6 

31.8 

2.6 

91.2 

8.8 

0.0 

C 

— 

— 

Voice Processing Management 

25.8 

62.3 

11.9 

24.0 

70.0 

6.0 

23.7 

74.2 

2.1 

Supervisory Skills 

56.7 

38.0 

5.3 

86.3 

12.7 

1.0 

C 

— 

— 

Support Systems Analysis 

* 

* 

* 

26.7 

63.4 

9.9 

24.7 

71.1 

4.1 

Project Management 

* 

* 

* 

47.5 

43.6 

8.9 

83.5 

15.5 

1.0 

Time Management 

* 

* 

* 

81.4 

17.6 

1.0 

93.8 

6.2 

0.0 

Work Flow Analysis 

* 

* 

* 

55.9 

41.2 

2.9 

83.3 

15.6 

1.0 

Hardware Configuration 

* 

* 

* 

17.8 

64.4 

17.8 

18.6 

74.2 

7.2 

Business Process Design 

* 

* 

* 

15.8 

72.3 

11.9 

10.4 

88.5 

1.0 

Operating Systems 

* 

* 

* 

50.0 

42.2 

7.8 

78.4 

16.5 

5.2 

Micrographics/Electronic Imaging 

* 

* 

* 

8.9 

71.3 

19.8 

7.2 

81.4 

11.3 

Team Building Skills 

* 

* 

* 

74.5 

24.5 

1.0 

86.6 

13.4 

0.0 


Note 1: Asterisks in some categories for Round One indicate competencies which did not appear in that round. These competencies 
were included in subsequent rounds as a result of Round One responses. 

Note II: “C” indicates the point where consensus (85 percent) was reached in the previous round and the element was dropped from 
subsequent rounds. 

VI = Very Important 
FI = Fairly Important 
NI = Not Important 
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frequencies for competencies for those who responded to all three rounds. 
These figures indicate that frequencies for competencies selected by respon¬ 
dents who replied to all three rounds of questionnaires appear to be fairly con¬ 
sistent with results for the whole group (those who responded to at least one 
round). 

Occupations 

Respondents were asked their opinions concerning various occupations for 
which a program graduate might qualify. There were two underlying reasons for 
doing this. First, the department was looking for some assurance that this pro¬ 
gram was being designed to fill occupational niches that actually exist. Second, 
since some of the members of several subgroups are potential employers, it was 
a good opportunity to encourage them to begin to think of the graduates as hav¬ 
ing the education and skills to fill positions in their organizations. Table 5 
shows the frequency responses in percentages during three rounds of questions 
on possible occupations for the new graduate. Respondents were asked to indi¬ 
cate whether a new graduate could, in their opinion, fill the suggested occupa¬ 
tions immediately after graduation or after acquiring some experience. In Round 
One respondents were given the option of answering either “Immediately,” 
With Experience,” or “Never.” “Never” was chosen for occupations which 
many respondents thought were inappropriate for these graduates. There was 
also a tendency to select “immediate” for these positions and then add a com¬ 
ment that it was really an inappropriate position for such a graduate. In subse¬ 
quent rounds, therefore, these occupations were dropped from the 
questionnaires; but others were added as a result of suggestions in Round One. 
Occupations which received a relatively high rating of “never” (over 15%) or 
“immediate” together with negative comments, and were therefore dropped, 
included receptionist, data processing operator, file clerk, and MIS manager. 
Comments suggested that the first three were rated “never” because the gradu¬ 
ate would be overqualified for such positions. The fourth was considered by 
respondents to be outside the sphere of expertise for graduates of this program. 

Although Table 5 indicates that many of the occupations could not be filled 
immediately by new program graduates, but only after some experience, a 
majority of respondents indicated that they would consider four co-op work 
terms sufficient experience for many of the occupations. In this instance, the 
occupations most frequently listed included trainer, administrative services 
supervisor, records analyst, and office automation analyst. Over 98% of the 
respondents supported the inclusion of some type of work experience in the pro¬ 
gram. Scheduled work terms were most often recommended. 
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Table 6 shows the frequency responses for occupations from all those who 
responded to all three rounds. As with competencies, there were no significant 
differences between the responses of this group and those of the whole group. 

Ninety-two percent of all respondents agreed that the emphasis of the pro¬ 
gram should move from office administration to information management. Of 
those who did not agree, many simply wanted another label instead of “informa¬ 
tion management,” although they were describing the same concept. 

Discussion of the Results 

While consensus was reached on several of the competencies and there was 
majority agreement on most, there were still others where very little agreement 
was reached. There was a fairly even split between people who considered 
equipment selection to be very important and those who considered it only fair¬ 
ly important. There was also substantial disagreement concerning the degree of 
importance of office environment planning, as well as desktop publishing, 
accounting software, systems analysis, and voice processing management. 
There could be several reasons for these results. In some instances, such as sys¬ 
tems analysis, comments were made that respondents did not have a clear 
understanding of what was involved in a particular competency. This was espe¬ 
cially true with the counsellors’ subgroup where the respondents repeatedly 
mentioned that they were not qualified or not sufficiently well informed to give 
an opinion. 

There was almost universal agreement concerning some competencies. 
These were generally areas in which high agreement was anticipated. For exam¬ 
ple, it was anticipated that both written and oral communication would be rated 
as very important. Somewhat surprisingly, keyboarding was rated as a very 
important element to be included in the program. Faculty had concluded that 
since keyboarding courses were offered in most high schools, it was no longer 
essential to include the teaching of such a skill at the postsecondary level. 
(Keyboarding is now planned as a short non-credit course for students who 
enter the program without this skill.) 

Throughout the three rounds there was movement of the whole group 
toward increasing the importance assigned to knowledge of software applica¬ 
tions such as word processing, spreadsheets, and databases. This was also true 
of the competencies of software evaluation, training skills, and telecommunica¬ 
tions. 

On the other hand, there was little movement throughout the three rounds 
in areas such as computer programming and research methods. While this was 


to 

Table 4 

Frequencies in Percentages for Competencies: Respondents to All Three Rounds 


Competencies 


R1 (N=61) 


R2 (N=61) 


R3 (N=61) 



VI 

FI 

NI 

VI 

FI 

NI 

VI 

FI 

NI 

Keyboarding 

78.7 

19.7 

1.6 

91.8 

6.6 

1.6 

C 

— 

— 

Proofreading 

82.0 

14.8 

3.3 

95.1 

1.6 

3.3 

C 

— 

— 

Written Communication Skills 

95.1 

4.9 

0.0 

96.7 

3.3 

0.0 

C 

— 

— 

Oral Communication 

90.2 

8.2 

1.6 

98.4 

1.6 

0.0 

C 

— 

— 

Records Management 

40.7 

57.6 

1.7 

36.7 

63.3 

0.0 

52.5 

47.5 

0.0 

Word Processing 

86.9 

11.5 

1.6 

98.4 

1.63 

0.0 

C 

— 

— 

Spreadsheets 

50.8 

44.3 

4.9 

85.2 

14.9 

0.0 

C 

— 

— 

Databases 

54.1 

44.3 

1.6 

93.4 

6.6 

0.0 

c 

— 

— 

Graphics 

36.1 

50.8 

13.1 

23.0 

77.0 

0.0 

24.6 

75.4 

0.0 

Accounting Software 

39.3 

54.1 

6.6 

32.8 

63.9 

3.3 

23.0 

75.4 

1.6 

Desktop Publishing 

31.1 

60.7 

8.2 

23.0 

75.4 

1.6 

27.9 

70.5 

1.6 

Decision Support Software 

* 

* 

* 

29.5 

59.0 

11.5 

13.1 

82.0 

4.9 

CAD/Engineering Software 

* 

* 

* 

18.3 

48.3 

33.3 

3.3 

62.3 

34.4 

Business Statistics 

9.8 

80.3 

9.8 

6.6 

86.9 

6.6 

13.1 

75.4 

11.5 

Research Methods 

13.1 

72.1 

14.8 

9.8 

82.0 

8.2 

11.5 

83.6 

4.9 

Telecommunications 

60.7 

37.7 

1.6 

88.5 

11.5 

0.0 

90.2 

9.8 

0.0 

Office Environment Planning/Design 

31.1 

60.7 

8.2 

21.3 

78.7 

0.0 

27.9 

70.5 

1.6 

Systems Analysis 

42.6 

47.5 

9.8 

13.1 

78.7 

8.2 

23.0 

72.1 

4.9 

Equipment Selection 

47.5 

39.3 

8.2 

32.8 

62.3 

4.9 

49.2 

45.9 

4.9 

Software Evaluation 

52.5 

39.3 

8.2 

90.2 

4.9 

4.9 

C 

— 

— 


Shirley Blair & Norman P. Uhl 



Table 4 (cont'd) 


Competencies 


R1 (N=151) 


R2 (N= 

= 102) 


R3 (N= 

=97) 


VI 

FI 

NI 

VI 

FI 

NI 

VI 

FI 

NI 

Training Skills 

59.0 

36.1 

4.9 

90.2 

8.2 

1.6 

C 

_ 

_ 

Computer Programming 

29.5 

41.0 

29.5 

11.5 

67.2 

21.3 

6.6 

72.1 

21.3 

Business Ethics 

62.3 

36.1 

1.6 

90.2 

9.8 

0.0 

C 

— 

— 

Voice Processing Management 

27.9 

65.6 

6.6 

27.9 

68.9 

3.3 

21.3 

78.7 

0.0 

Supervisory Skills 

55.7 

39.3 

4.9 

86.9 

13.1 

0.0 

86.9 

13.1 

0.0 

Support Systems Analysis 

* 

♦ 

* 

23.0 

67.2 

9.8 

29.5 

70.5 

0.0 

Project Management 

* 

* 

* 

47.5 

42.6 

9.8 

82.0 

16.4 

1.6 

Time Management 

* 

* 

* 

85.2 

14.8 

0.0 

96.7 

3.3 

0.0 

Work Flow Analysis 

* 

* 

* 

49.2 

47.5 

3.3 

86.7 

11.7 

1.7 

Hardware Configuration 

* 

* 

* 

19.77 

63.9 

16.4 

18.0 

78.7 

3.3 

Business Process Design 

* 

* 

* 

13.1 

77.0 

9.8 

8.3 

90.0 

1.7 

Operating Systems 

* 

* 

* 

50.8 

39.3 

9.8 

86.9 

11.5 

1.6 

Micrographics/Electronic Imaging 

* 

* 

* 

8.2 

73.8 

18.0 

4.9 

88.5 

6.6 

Team Building Skills 

* 

* 

* 

77.0 

23.0 

0.0 

86.9 

13.1 

0.0 


Note 1: Asterisks in some categories for Round One indicate competencies which did not appear in that round. These competencies 
were included in subsequent rounds as a result of Round One responses. 

Note II: “C” indicates the point where consensus (85 percent) was reached in the previous round and the element was dropped from 
subsequent rounds. 

VI = Very Important 
FI = Fairly Important 
NI = Not Important 
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Table 5 

Frequencies in Percentages for Occupations: All Respondents 


to 

to 


Occupations 

Round 1 (N=151) 

Round 2 (N= 102) 

Round 3 (N=97) 

I 

E 

N 

1 

E 

I 

E 

Administrative Assistant 

50.3 

48.3 

1.3 

92.1 

7.9 

C 


WP Supervisor 

17.9 

81.5 

0.7 

13.9 

86.1 

— 

C 

LAN Manager 

6.6 

84.8 

8.6 

9.0 

91.0 

- 

C 

WP Operator 

82.8 

2.6 

14.6 

100.0 

0.0 

c 

- 

Trainer 

19.5 

78.5 

2.0 

16.8 

83.2 

10.3 

89.7 

Training Manager 

2.7 

91.9 

5.4 

5.1 

94.9 

- 

C 

Systems Analyst 

16.9 

66.2 

16.9 

4.0 

96.0 

- 

C 

Records Manager 

47.0 

50.3 

2.6 

27.7 

72.3 

14.4 

85.6 

Office Manager 

19.3 

78.7 

2.0 

13.0 

87.0 

- 

C 

Information Manager 

18.0 

75.3 

6.7 

11.1 

88.9 

- 

C 

Records Analyst 

++ 

++ 

++ 

41.2 

58.8 

13.4 

86.6 

Admin. Services Supervisor 

++ 

++ 

++ 

21.2 

78.8 

9.3 

90.7 

Business Procedures Analyst 

++ 

++ 

++ 

32.7 

67.3 

10.3 

89.7 

Office Automation Analyst 

++ 

++ 

++ 

42.9 

57.1 

10.3 

89.7 

Information Specialist 

++ 

++ 

++ 

48.0 

52.0 

11.3 

88.7 

Jr. Records Mgt. Consultant 

++ 

++ 

++ 

76.0 

24.0 

30.9 

69.1 


"I" signifies that the new program graduate could enter this occupation immediately after graduation 
"E" signifies that some experience would be necessary before the graduate could qualify for this position. 
"N" signifies "never" and was used only in Round One in order to eliminate inaapropriate occupations. 
"++" signifies that this occupation was added after Round One following suggestions from respondents. 
"C" signifies that concensus (85 percent) was reached in the previous round. 
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Table 6 

Frequencies in Percentages for Occupations: Respondents to All Three Rounds 



Round 1 (N=61) 

Round 2 (N=61) 

Round 3 (N=61) 

Occupations 

I 

E 

N 

I 

E 

I 

E 

Administrative Assistant 

57.6 

39.0 

3.4 

93.4 

6.6 

C 

_ 

WP Supervisor 

19.7 

76.7 

1.6 

10.0 

90.0 

-- 

C 

LAN Manager 

6.6 

85.2 

8.2 

6.7 

93.3 

- 

C 

WP Operator 

86.9 

0.0 

13.1 

100.0 

0.0 

c 

- 

Trainer 

20.0 

76.7 

3.3 

9.8 

90.2 

14.8 

85.2 

Training Manager 

3.3 

88.3 

8.3 

3.3 

96.7 

- 

C 

Systems Analyst 

18.6 

69.5 

11.9 

1.7 

98.3 

-- 

C 

Records Manager 

47.5 

47.5 

4.9 

19.7 

80.3 

14.8 

85.2 

Office Manager 

25.0 

70.0 

5.0 

8.2 

91.8 

- 

C 

Information Manager 

19.7 

72.1 

8.2 

5.0 

95.0 

- 

C 

Records Analyst 

++ 

++ 

+4- 

36.2 

63.8 

9.8 

90.2 

Admin. Services Supervisor 

++ 

++ 

+4- 

16.7 

83.3 

11.8 

88.5 

Business Procedures Analyst 

++ 

++ 

++ 

23.7 

76.3 

9.8 

90.2 

Office Automation Analyst 

++ 

++ 

4-4- 

35.6 

64.4 

6.6 

93.4 

Information Specialist 

++ 

++ 

4-4- 

39.0 

61.0 

14.8 

85.2 

Jr. Records Mgt. Consultant 

++ 

++ 

4-4- 

78.3 

21.7 

32.8 

67.2 


"I" signifies that the new program graduate could enter this occupation immediately after graduation 
"E" signifies that some experience would be necessary before the graduate could qualify for this position. 
"N" signifies "never" and was used only in Round One in order to eliminate inaapropriate occupations. 
"++" signifies that this occupation was added after Round One following suggestions from respondents. 
"C" signifies that concensus (85 percent) was reached in the previous round. 
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not unexpected when one considers that the graduates will be prepared to be 
end-user specialists rather than software developers, it is surprising that certain 
other areas were not assigned a higher degree of importance. For example, only 
slightly fewer than 51.5% of third-round respondents selected records manage¬ 
ment as being very important, and 48.5% thought it was only fairly important. 
(At the same time, over 85% felt that graduates with some experience could fill 
positions as records managers.) Since this is an area which is generally consid¬ 
ered by those in the field of information management to be an integral part of 
information management, it was expected to receive a higher rating, particularly 
when comments were included on the questionnaire in the final round suggest¬ 
ing that it might be of greater importance than was previously assigned by 
respondents. 

Voice processing management, despite the growing importance of voice 
mail in organizations and lack of training available, was selected as very impor¬ 
tant by only 23.7% of respondents in Round Three. It is possible, however, that 
this also indicates an imcomplete understanding of the area. One potentially 
valuable benefit of this study has been the development of good rapport with 
many respondents, and people in industry in particular. Many respondents took 
sufficient interest in the project that they wrote frequent comments and sugges¬ 
tions and were prepared to spend a significant amount of time completing ques¬ 
tionnaires. 

A considerable amount of information concerning basic components of the 
new courses has been acquired. While it would have been preferable to have a 
larger sample from industry, nevertheless the general feedback has been, for the 
most part, informed and enthusiastic. Some skills components, such as key¬ 
boarding and word processing, have appeared as having greater importance than 
was envisioned in the revised program. Confirmation has been received that 
excellent oral and written communication skills are essential for the new gradu¬ 
ate and that these competencies must play a major role in the curriculum. 
Knowledge of software applications such as spreadsheets and databases was 
emphasized, though, surprisingly, graphics and desktop publishing were not. 
Possibly, these areas are still restricted to specialized types of work and only in 
select organizations. The high importance attached to elements such as team¬ 
building skills, project management, and time management suggests that “peo¬ 
ple skills” are becoming more important in the workplace and should be incor¬ 
porated in the program. 

The low group rating for records management was of some concern. The 
present intention is to incorporate a minimum of four courses in records 
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management in the new information management program. It is possible that 
the low rating from many respondents could indicate a misinterpretation of what 
the field includes. There is a tendency in some areas of business to think of 
records management as “filing.” 

Information received concerning occupations has helped to give some com¬ 
fort to the department that new graduates will find employment in a variety of 
areas and that there is a real need for individuals with such specialized educa¬ 
tion. While there is still some resistance to change from the present program on 
the part of some subgroups, generally the results indicate that the direction in 
which this department is moving is appropriate for the conditions and situations 
in today’s business world. 


Recommendations 

Further inquiry is needed into the appropriateness of including a large concen¬ 
tration of records management in an information management program. There is 
a need to confirm that the program is serving regional business needs by offer¬ 
ing these courses. If the new program is to include a large component of records 
management, the department might be well advised to include some more 
detailed descriptions of the records management courses in its promotional liter¬ 
ature. 

A need has been identified for greater promotion of the program in order 
better to inform various groups of what the program is, and will be, offering. 

It was recommended that, since the direction in which this department is 
moving is unique in the particular geographic area, a follow-up study should be 
done starting from the time that the first graduates of this program begin 
employment. This could provide much valuable feedback both to the university, 
employers, and future students. 

Discussion of the Delphi Technique 

What is an appropriate response rate? This is a common question asked of 
Delphi studies and there is no simple answer. Excellent response rates are possi¬ 
ble by providing the appropriate incentive. For example, in one study 
(Uhl, 1971) 305 of 326 university students participated in all three rounds. Each 
student received a cheque upon completion of all rounds. Clearly this type of 
incentive is not always possible, nor is a 94% rate of return always necessary. It 
depends on the objectives of the study. For example, in the study reported here, 
five groups were included. Different response rates were expected for these dif¬ 
ferent groups. On the first questionnaire it was expected that over 70% of the 
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present students and the Advisory Board would participate (85% and 75% 
participated). In contrast, the first questionnaire was sent to a large number of 
graduates because it was expected that many would be returned because of inac¬ 
curate mailing addresses. Another group consisted of school counsellors. The 
purpose of including school counselors was to introduce them to the new cur¬ 
riculum so they could be more knowledgeable in advising high school students. 
Thus, as many counsellors as feasible were selected. When only 17% completed 
the first questionnaire, the objective for this group was clearly not met, and 
action was taken to increase this group’s participation. The objective was not to 
obtain a more representative sample, but to educate more counselors. One must 
also consider the attrition from one round to the next. If the study is of interest, 
there will be less attrition between rounds than on the first round. However, 
other factors may influence participation. In this study, a number of the business 
people were transferred or given different responsibilities as their company 
made adjustments to the hard economic times. The attrition from round to round 
should be of concern because some people could drop out because they disagree 
with the majority. If this is the situation, consensus may occur only because the 
people who agree remain in the study. This should be checked by comparing the 
first questionnaire responses of those who remain in the study with those who 
drop out. 

There are many other questions concerned with administering the Delphi 
technique. For those interested in considering this technique, Uhl (1983) pre¬ 
sents some initial planning guidelines as well as a discussion of the major steps 
in conducting such a study. The initial planning guidelines as stated by Uhl are: 

1. Be certain there is not a simpler and less costly procedure that 
will achieve the same results. 

2. Do not underestimate the time and resources needed to provide 
the necessary feedback. 

3. Select the participants with care and in a manner consistent with 
the objectives of the study. 

4. Use unambiuguous statements, neither too vague nor too specif¬ 
ic, to obtain valid and efficient input from participants. 

5. Do not mislead the participants. 

6. Do not ignore the responses of participants who disagree with 
the majority. 
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7. Do not impose the views and biases of the study designers on 
the participants. 

8. Allow enough time between rounds to prepare and distribute 
feedback, but do not allow so much time that participants lose 
interest in the study. 

9. Provide participants with enough incentive to remain interested 
throughout the study (p.88). 

The major steps in a Delphi study involve identifying the project’s pur¬ 
pose^), establishing a project committee, selecting the participants, determining 
the specific methodology, developing the round one questionnaire, analyzing 
the round one content, processing round two, processing round three and suc¬ 
ceeding rounds, and using the results. Each of these is discussed in detail in Uhl 
(1983). 


Conclusions 

The above is an illustration of using the Delphi process for curriculum improve¬ 
ment. It illustrates some advantages of the process. For example, it is possible, 
for the cost of postage and copying, to include quite diverse groups with differ¬ 
ing degrees of expertise. The participants are free to select a convenient time to 
complete the questionnaires. Participants are provided feedback as part of the 
process and become more knowledgeable about the program. They also experi¬ 
ence a sense of having assisted in developing the program and thus acquire an 
interest in the program. Many who participate indicate a willingness to provide 
further assistance, if necessary. An important by-product is the publicity given 
the program. Finally, in the frequent cases when convergence does not occur, 
groups which disagree and their reasons for disagreeing can be identified. 
Further education can be provided to these groups, if deemed necessary; or at 
the very least, a better understanding of their reasons for disagreement is 
obtained. 

There are also some disadvantages to the Delphi technique. Since there are 
at least three rounds of questions, a person must have some interest in the topic 
to participate. A lack of interest can lead to low participation rates, as illustrated 
in this study. The best way to increase this rate is through an effective cover let¬ 
ter and at least one follow-up. Another disadvantage is that, as in this study, a 
few questions can remain unanswered. When this occurs, it must be determined 
whether the answers are worth another round of questions for all groups. In this 
particular study, special follow-up was done with certain groups to obtain these 
answers, rather than conducting another round of questions for all groups. 
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It is also imperative for any Delphi process that the participants trust the 
moderator. Individual responses must be kept anonymous and feedback must be 
accurate. The moderator must be given sufficient time to perform the task. 

While curriculum improvement through the Delphi technique has not been 
employed frequently in Canada, it is hoped that this technique will prove useful 
as greater emphasis is placed on quality programs at all levels. 
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